Distributed cross-platform user interface and application projection

ABSTRACT

A mobile device application adapted to provide multimedia content to a target canvas is described. The application includes sets of instructions for: establishing a communication link with an external system associated with the target canvas; rendering multimedia content for playback by the target canvas; and sending the rendered multimedia content to the target canvas over the communication link. A mobile device application adapted to execute web-based applications in a browser associated with an external system includes sets of instructions for: establishing a communication link with the external system; accessing a web-based application; and rendering content associated with the application and sending the rendered content to the external system for display in the browser. A system adapted to provide multimedia content includes: a target adapted to display multimedia content; a host adapted to generate multimedia content; and a remote server adapted to at least partially control the display of multimedia content.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationSer. No. 61/733,367, filed on Dec. 4, 2012.

BACKGROUND OF THE INVENTION

Mobile devices (e.g., smartphones, tablet devices, laptop PCs, etc.)that are able to access one or more networks (e.g., a cellular network,a local area wireless network, the Internet, etc.) are ubiquitous insociety. Such devices may be capable of processing, displaying, and/orotherwise interacting with multimedia elements (e.g., video, photo,audio, etc.) utilizing various local and/or web-based applications.

Many typical systems and/or devices (e.g., in-vehicle systems includingdisplay screens, medical systems, televisions (TVs), etc.) may becapable of processing, presenting, and/or otherwise interacting withmultimedia elements. Such devices may not have access to externalnetworks. For instance, a head unit (HU) multimedia element may be ableto display multimedia elements such as video from a local source (e.g.,a DVD player connected to the HU). In addition, in cases where such anelement is able to access an external network, the access may be limitedby the software available to the local element (e.g., even an elementthat may be able to access a network to retrieve, for instance, mapdata, may not be able to provide a web browser).

Such systems and/or devices may include limited processing capability(especially as compared to a mobile device). Furthermore, these systemsand/or devices may execute device-specific software that may not beupdated as frequently (or as easily) as software updates are madeavailable for a typical mobile device. Likewise, upgrading such systemsor devices may be impractical or impossible due to limitations imposedby a physical housing, available power sources, communicationsconnections, etc.

Therefore there exists a need for a solution that is able to provide aneasily updated or upgraded approach that utilizes the processing powerof mobile devices to provide extended and improved network access,functionality and multimedia performance to existing systems.

BRIEF SUMMARY OF THE INVENTION

Some embodiments allow external systems or devices connected to hostdevices such as smartphones to easily run third party applicationsdeveloped using various appropriate web technologies (e.g., HTML5, ascripting language such as JavaScript, cascading style sheets (CSS),etc.) and/or native mobile technologies (e.g., Objective-C, Java, C #,etc.). Such external systems or devices may include, for example,in-vehicle head-unit (HU) devices, medical devices, TVs, etc. Someembodiments allow these external systems or devices to be kept simple,low cost and be extensible using third party applications.

Some embodiments include a set of components that are built on top ofcore mobile communication stack (or “gateway” or “interface”) transportand HTTP layers. The performance and the flexibility of the gateway maybe utilized and extended it with configurable infrastructure that allowsdownloading, managing and serving applications to connected target (or“client”) elements. Some embodiments may be configured to supportdifferent application delivery and rendering options—from deliveringHTML code directly to the target, to rendering on the host anddelivering rendered screens to the target. Such an approach accommodatesdifferent devices having different capabilities. Some embodimentsprovide a common programming interface and infrastructure that allowsdevelopers to generate applications that are able to be executed usingdifferent targets.

The solution of some embodiments may be extensible (new components andfeatures may be added without changing the main architecture),configurable (the components may be easily setup for differentconfigurations), operating system (OS) independent (may run on all majoroperating systems), bandwidth efficient (the solution may utilizecompression, encoding, and/or other appropriate techniques to minimizebandwidth usage), and responsive (the solution may use efficientprotocols to create a seamless user experience on the client device). Inaddition, the solution may allow application portability, whereapplications written for one operating system may run on another or on aclient with minimal change.

A first exemplary embodiment of the invention provides a mobile deviceapplication adapted to provide multimedia content to a target canvas.The application includes sets of instructions for: establishing acommunication link with an external system associated with the targetcanvas; rendering multimedia content for playback by the target canvas;and sending the rendered multimedia content to the target canvas overthe communication link.

A second exemplary embodiment of the invention provides a mobile deviceapplication adapted to execute web-based applications in a web browserassociated with an external system. The application includes sets ofinstructions for: establishing a communication link with the externalsystem; accessing a particular web-based application; and renderingcontent associated with the particular web-based application and sendingthe rendered content to the external system for display.

A third exemplary embodiment of the invention provides a system adaptedto provide multimedia content to a user. The system includes: a targetadapted to display multimedia content; a host adapted to generatemultimedia content for display by the target; and a remote serveradapted to interact with the host to at least partially control thedisplay of multimedia content.

The preceding Summary is intended to serve as a brief introduction tovarious features of some exemplary embodiments of the invention. Otherembodiments may be implemented in other specific forms without departingfrom the spirit of the invention.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The novel features of the invention are set forth in the appendedclaims. However, for purpose of explanation, several embodiments of theinvention are set forth in the following drawings.

FIG. 1 illustrates a schematic block diagram of a conceptual system ofsome embodiments;

FIG. 2 illustrates a schematic block diagram of connection schemes thatmay be used by some embodiments of the system;

FIG. 3 illustrates a schematic block diagram of a host element of someembodiments;

FIG. 4 illustrates a schematic block diagram of a target element of someembodiments;

FIG. 5 illustrates a schematic block diagram of a browser-based systemused by some embodiments to provide services to a target I/O elementcapable of executing a web browser;

FIG. 6 illustrates a schematic block diagram of a screen projectionsystem used by some embodiments to provide services to a target I/Oelement capable of providing multimedia content;

FIG. 7 illustrates a schematic block diagram of a vector screenprojection system used by some embodiments to provide services to atarget I/O element capable of providing multimedia content;

FIG. 8 illustrates a schematic block diagram of a system of someembodiments that utilizes hardware encoding and/or decoding to provideoptimized services to a target I/O element capable of providingmultimedia content;

FIG. 9 illustrates a data flow diagram of a system including variousdata elements and pathways that may be used by some embodiments;

FIG. 10 illustrates a schematic block diagram of a back-end system usedby some embodiments to manage host applications in some embodiments;

FIG. 11 illustrates a flow chart of a host-side (or “server-side”)process used by some embodiments to allow interaction among at least onehost and at least one target;

FIG. 12 illustrates a flow chart of a target-side (or “client-side”)process used by some embodiments to allow interaction among at least onehost and at least one target;

FIG. 13 illustrates a flow chart of a process used by some embodimentsto encode data to be sent from a host to a target;

FIG. 14 illustrates a flow chart of a conceptual process used by someembodiments to adaptively adjust various attributes to achieve optimizedthroughput and quality based on current operating conditions;

FIG. 15 illustrates a flow chart of a target-side conceptual processused by some embodiments to capture and react to user input events;

FIG. 16 illustrates a flow chart of a host-side conceptual process usedby some embodiments to capture and react to user input events; and

FIG. 17 conceptually illustrates a schematic block diagram of a computersystem with which some embodiments of the invention may be implemented.

DETAILED DESCRIPTION OF THE INVENTION

The following detailed description is of the best currently contemplatedmodes of carrying out exemplary embodiments of the invention. Thedescription is not to be taken in a limiting sense, but is made merelyfor the purpose of illustrating the general principles of the invention,as the scope of the invention is best defined by the appended claims.

Various inventive features are described below that can each be usedindependently of one another or in combination with other features.

Broadly, embodiments of the present invention generally provide a set ofserver, host elements (e.g., smartphones, tablet devices, etc.) andtarget elements (e.g., vehicle head units, TVs, medical devices, etc.)that can be used and configured in various appropriate ways. Suchcomponents will be described in more detail in Section I below.

In a first configuration, some embodiments may use one or more targetelements to host a web browser. Such an approach may allow webapplication to be stored locally by the target elements (or “client”),allowing the applications to be used even without any networkconnectivity. In addition, such a configuration may allow for fastresponse as the applications are local and may thus be rendered quickly.This approach may require the target element to have relatively powerfulhardware in order to render the applications. In addition, any browserused by the target device may have to be updated regularly to maintaincompatibility with evolving technology. In some cases (e.g., vehiclehead units) appropriate updates may not be available to the targetdevice and/or may not be able to be installed easily by a user.

In a second configuration, a mobile device may host and render anapplication and the target element may act as a screen projectionclient. Such rendering may be performed off-screen by the mobile deviceusing the screen resolution of the client device. Such an approachallows for a simple, cheap and easy to maintain target device becausethe mobile device provides most of the required processing power. Inaddition, updates to mobile device browser software may be providedregularly and may be installed easily (or may be updated automatically).This approach may also allow standard technologies such as virtualnetwork computing (VNC), standard video encoding formats, etc. to beused. In some cases, this configuration may have a slower response asprojection screens (and/or other appropriate data) may be sent over apotentially slower link (e.g., an in-vehicle communications bus).

In a third configuration, a mobile device may similarly host anapplication, but may prepare and send data in vector format to thetarget element, which may act as a screen projection client. The clientmay maintain a cache of common resources, thus allowing for improvedrendering and reduced traffic among the mobile device and the clientdevice, providing improved performance. Such an approach allows for arelatively simple, cheap and easy to maintain target device because themobile device provides most of the required processing power. Inaddition, updates to mobile device browser software may be providedregularly and may be installed easily (or may be updated automatically).

Although the various configurations have been separated logically forsimplicity, one of ordinary skill in the art will recognize that thevarious configurations (or “modes”) may be combined to provide supportof multiple modes. Thus, depending on the capabilities of the hostand/or target, the host may send full screens or vector commands, asappropriate. In addition, other technologies, such as screencasttechnologies, may also be used in place of VNC or the vector approachdescribed above.

Another implementation may allow smartphone applications to use nativecontrols with or without a web view. The host may prepare an off-screenbuffer with the size and resolution of the client and use the buffer torender the user interface (UI) and simulate the user input events to theinterface.

Several more detailed embodiments of the invention are described in thesections below. Section I provides a conceptual description of thehardware architecture of some embodiments. Section II then describes aconceptual software architecture of some embodiments. Next, Section IIIprovides a conceptual description of application and policy managementprovided by some embodiments. Section IV then describes variousalgorithms used by some embodiments. Section V follows with adescription of a conceptual computer system which implements some of theembodiments of the invention.

I. Hardware Architecture

Sub-section I.A provides a conceptual description of a system of someembodiments. Sub-section I.B then describes a conceptual control elementof some embodiments. Lastly, sub-section I.C describes a conceptualtarget element of some embodiments.

A. System

FIG. 1 illustrates a schematic block diagram of a conceptual system 100of some embodiments. Specifically, this figure shows the variouscommunication pathways among the system elements. As shown, the systemmay include a target system or device 110 which may include one or moreinput/output (I/O) elements 120 and a communication bus 130, one or morehost systems or devices 140, one or more networks 150, a set of remoteservers 160 with associated storage(s) 170, and a set of third-partyservers 180 with associated storage(s) 190.

Target element 110 may be any set of systems, devices, components,and/or other appropriate elements that may be able to communicate with ahost element 140. Each of the I/O elements 120 may include variousappropriate components (e.g., a display screen, an audio output, a setof user controls, a touchscreen, etc.) and/or capabilities (e.g.,encoding or decoding of audio video and/or other multimedia,communicating across one or more networks, etc.). The term “target” maybe used throughout to indicate a target element or set of I/O elements,as appropriate. The communication bus 130 may include various hardwareand/or software elements, as appropriate. Such a bus may be wired orwireless and may allow various components of the target element 110, thehost element 140, and/or other components to communicate.

In some embodiments, the target element 110 may be a vehicle (e.g., acar, airplane, bus, train, recreational vehicle (RV), etc.). Such asystem may include I/O elements 120 such as head units, monitors, TVs,video screens, audio elements (e.g., speakers, headphone jacks, etc.),user interfaces (e.g., control buttons, touchscreens, microphones,etc.), audio players (e.g., a stereo system), and/or other appropriateelements.

Each host element 140 may be an electronic system or device that iscable of interacting with the target 110 (e.g., a smartphone, a tabletdevice, a notebook computer, a dedicated media player, etc.). The host140 may interact with one or more I/O elements 120 included in thetarget system 110, as appropriate. For instance, in some embodiments asmartphone host 140 may send multimedia content to multiple vehicle headunit I/O elements 120. In addition, as described below in reference toFIG. 2, system 100 may include multiple targets 110 and/or hosts 140, asappropriate.

The network 150 may include one or more communication networksaccessible to the host 140 (and/or the target 110). Such networks mayinclude, for instance, cellular communication networks, wirelessnetworks, wired networks, networks of networks (e.g., the Internet),and/or other appropriate networks. Such networks may be accessed usingvarious appropriate combinations of hardware and/or software elements,as appropriate.

The remote server 160 may include one or more devices that are able toaccess the network 150 and thus communicate with the host 140. Storage170 may include one or more devices accessible to the remote server 160that are able to store data and/or instructions. In some embodiments,the remote server 160 may execute software that is adapted tocommunicate with associated software executed by the host 140, target110, and/or other appropriate elements.

The third-party server 180 may include one or more devices that are ableto access the network 150 and thus communicate with the host 140.Storage 190 may include one or more devices accessible to thethird-party server 180 that are able to store data and/or instructions.In some embodiments, the third-party server and/or storages 190 may beaccessed using one or more APIs.

One of ordinary skill in the art will recognize that system 100 isconceptual in nature and may be implemented in various different wayswithout departing from the spirit of the invention. For instance,different embodiments may include different specific components and/orcommunication pathways among components. As another example, someembodiments may include addition components or fewer components thanshown (e.g., some embodiments may operate without access to any networksor servers).

FIG. 2 illustrates a schematic block diagram of example connectionschemes 200-210 that may be used by some embodiments of the system 100.Specifically, this figure shows multiple-target and multiple-hostconfigurations that may be used.

As shown, a first example configuration 200 may include multiple targets220 communicating with a single host 230. Such a configuration may beused, for example, to stream media from a smartphone to multiple TVs.

In a second example configuration 210 may include a target 240 withmultiple I/O elements 245-265 and multiple hosts 270-280. In thisexample, a first host 270 may interact with multiple I/O elements245-250 (e.g., by sending multimedia content to multiple vehicle headunits and/or displays). Another I/O element 255 may not interact withany host (e.g., when the I/O element is running a native applicationsuch as a navigation application in a vehicle). A second host 275 mayinteract with a single I/O element 260. Additional hosts 280 and/or I/Oelements 265 may each be configured in various appropriate ways (e.g.,one host associated with one I/O element, one host associated withmultiple I/O elements, an I/O element with no associated host, etc.).

In addition to the examples shown in FIG. 2, one of ordinary skill inthe art will recognize that various other appropriate schemes may beused, as appropriate. For instance, multiple hosts may interact withmultiple targets, where each target may include multiple distinct I/Oelements that are each able to interact (or not) with one of the hosts.As another example, in some embodiments multiple hosts may interact witheach I/O element (e.g., a first host may transmit video data to a set ofI/O elements and a second host may transmit audio data to another,overlapping, set of I/O elements).

B. Host

FIG. 3 illustrates a schematic block diagram of a host element 300 ofsome embodiments. Specifically, this figure shows various conceptualsub-elements and communication pathways that may be used by the host. Asshown, the host system or device 300 may include one or more processors310, one or more hardware encoders 320, a set of I/O elements 330, oneor more communication modules 340, and one or more storages 350.Examples of such host elements include mobile phones, smartphones, PCs,tablet devices, dedicated media players (e.g., DVD players, Blu-rayplayers, CD players, mp3 players, etc.), and/or other appropriate setsof components.

Each processor 310 may be able to perform various operations using setsof data and/or instructions. Each hardware encoder 320 may bespecifically adapted to process certain types of inputs (e.g., videoinput from a camera, audio input from a microphone, etc.) and generatedata in an appropriate format for use by another multimedia element.Each I/O element 330 may be adapted to receive input from a user and/orprovide output to a user (e.g., a touchscreen, keypad, mouse and/orother cursor control device, display screen, speaker, microphone, etc.).Each communication module 340 may be adapted to communicate with one ormore other systems and/or devices across various available pathways(e.g., wired connections, wireless connections, network connections,etc.) using various appropriate protocols (e.g., Bluetooth, IEEE 802.11,etc.). Each storage 350 may be adapted to store data and/orinstructions. The various components 310-350 may be able to communicateamongst each other using various local pathways such as busses,networks, etc.

Although the host 300 may conceptually be considered as a single systemor device, in some embodiments the host may include various combinationsof distinct elements (e.g., a smartphone with externalmicrophone/headphones, a tablet device with external speakers, etc.).Such components may be connected using various physical elements (e.g.,cables, connectors, receptacles, etc.) and/or wireless or virtualelements.

The host of some embodiments may include various UI elements (e.g.,touch screens, buttons, keypads, etc.) that may be utilized in variousappropriate ways. For instance, some embodiments may allow a user to atleast partially control a target element using various input elementsprovided by the host (e.g., control features on a touchscreen, volumecontrol buttons, etc.). In addition, a component such as a displayelement (e.g., a touchscreen, an indicator light, etc.) may be used toprovide information to a user.

One of ordinary skill in the art will recognize that host 300 isconceptual in nature and may be implemented in various different wayswithout departing from the spirit of the invention. For instance,different embodiments may include different specific components and/orcommunication pathways among components. As another example, someembodiments may include addition components or fewer components thanshown (e.g., some embodiments may not include hardware encoders).

C. Target

FIG. 4 illustrates a schematic block diagram of a target element 400 ofsome embodiments. Specifically, this figure shows various conceptualsub-elements and communication pathways that may be used by the target.As shown, the target system or device 400 may include one or moreprocessors 410, one or more hardware decoders 420, a set of UI elements430, a set of multimedia I/O elements 440, one or more communicationmodules 450, and one or more storages 360. Examples of such targetelements include TVs, vehicle HUs, display monitors, smartphones, PCs,tablet devices, and/or other appropriate sets of components.

Each processor 410 may be able to perform various operations using setsof data and/or instructions. Each hardware decoder 420 may bespecifically adapted to process certain types of received data (e.g.,video, audio, etc.) and generate appropriate outputs that are optimizedfor various media presentation elements (e.g., display screens, audiooutputs, etc.). Each UI element 430 may be adapted to receive inputsfrom and/or send outputs to a user (e.g., a touchscreen, keypad,buttons, knobs, mouse and/or other cursor control device, displayscreen, speaker, microphone, etc.). Each multimedia I/O element 440 maybe adapted to provide multimedia content to a user (e.g., a videoscreen, a set of speakers, etc.). Each communication module 450 may beadapted to communicate with one or more other systems and/or devicesacross various available pathways (e.g., wired connections, wirelessconnections, network connections, etc.) using various appropriateprotocols (e.g., Bluetooth, IEEE 802.11, etc.). Each storage 460 may beadapted to store data and/or instructions. The various components410-460 may be able to communicate amongst each other using variouslocal pathways such as busses (e.g., a controller area network (CAN)bus), networks, etc.

Although the target 400 may conceptually be considered as a singlesystem or device, in some embodiments the target may include variouscombinations of distinct elements (e.g., a display screen with aheadphone jack, a tablet device with external speakers, etc.). Suchcomponents may be connected using various physical elements (e.g.,cables, connectors, receptacles, etc.) and/or wireless or virtualelements.

One of ordinary skill in the art will recognize that target 400 isconceptual in nature and may be implemented in various different wayswithout departing from the spirit of the invention. For instance,different embodiments may include different specific components and/orcommunication pathways among components. As another example, someembodiments may include addition components or fewer components thanshown (e.g., some embodiments may not include hardware decoders).

II. Software Architecture

Sub-section II.A provides a conceptual description of a browser-basedconfiguration of some embodiments. Sub-section II.B then describes ascreen projection implementation of some embodiments. Next, sub-sectionII.C describes vector screen projection implementation of someembodiments. Sub-section II.D then describes hardware encoding anddecoding utilized by some embodiments. Next, sub-section II.E describesa conceptual data flow and data elements used by some embodiments.Lastly, sub-section II.F describes an exemplary set of protocol commandsthat may be used by some embodiments.

A. Browser

Some embodiments may be implemented using a browser that is able to beexecuted by the target system or device. In this configuration, thetarget browser may be used to render and execute third-partyapplications on a target I/O element.

FIG. 5 illustrates a schematic block diagram of a browser-based system500 used by some embodiments to provide services to a target I/O elementcapable of executing a web browser. Specifically, this figure showsvarious software components that may be used by the various systemelements. Such a software system may be implemented using the hardwaresystem 100 described above. As shown, the target I/O element (or system)120 may execute a web browser host 510 with an associated API 515, a setof third-party web applications 520, a command interface 530, anapplication manager 540, and local data 550. The host element 140 mayexecute a gateway 560. Each third-party server 180 may provide aninterface 570 and various third-party data 580. The remote server 160 ofsome embodiments may provide an application manager 590 and globalapplication data 595.

The web browser 510 may be any appropriate browser that is able to beexecuted by the target device. The API 515 may be, for example, aplug-in based (e.g., JavaScript) API that may allow third-party webapplications to communicate with the application manager 540 and withthe target system in a uniform way. The API 515 may be modular and maybe based on the security policies defined by the application manager 540for each application. Thus, the API 515 may restrict elements eachapplication is able to access and/or operations the application is ableto perform.

Each third-party web application 520 may be an application adapted tooperate using the web browser host 510. The command interface 530 mayallow the target I/O element 120 to interact with the gateway 560provided by the host 140, and thus also the third-party servers 180and/or remote servers 160. The third-party interface 570 may allow thegateway 560 (and thus also the target element 120 and remote server 160)to retrieve application data (and/or other types of data and/orinstructions) from the third-party data 580.

In the various architectures described throughout, the gateway 560and/or interface 530 of some embodiments may be used as the underlyingcommunication link between the target (e.g., a HU) and the host (e.g., asmartphone). Such a communication link allows for a uniformcommunication interface for different physical links (e.g., Bluetooth,USB, WiFi, future link types, etc.).

The application manager 540 may be a locally-executed target componentthat manages third-party web applications 520. Such management mayinclude lifecycle, software updates, and/or permissions via manifestfiles. The application manager 540 may manage local data 550 where thethird-party applications 520 are stored. The application manager 540 maycommunicate with the application manager service 590 to obtain thelatest updates and manifest changes. The application manager 540 may, insome embodiments, enforce execution policies based at least partly ontarget system status (e.g., whether a vehicle is moving, stopped,parked, etc.) for a safer driving experience.

Applications may be updated from the remote server 160 (using anapplication manager server). The applications 520 may be rendered by thelocal web browser host 510, which exposes an API library associated withthe API 515 that allows the third party applications to communicate withthe application manager 540 and/or other target services in a secure andcontrolled fashion. The target may communicate to the internet via thecommand interface 530 and/or gateway 560. Third party applications 520may communicate with associated web servers via the gateway 560 orcommunicate with mobile device applications via an application routerinfrastructure of some embodiments.

The application manager 590 may provide a web service API to the localapplication manager 540 to query for application manifests, newapplications, and/or application updates. The application manager 590may allow clients to query for available applications, downloadapplications, and/or update applications. The application manager maymanage the global application data 595. The global application data 595may include all available third-party applications. The applications maybe cataloged by type, target element model, etc. Each application mayhave an associated manifest file which contains security policies.

One of ordinary skill in the art will recognize that system 500 may beimplemented in various appropriate ways without departing from thespirit of the invention. For instance, different embodiments may includedifferent specific components. As another example, some embodiments mayinclude more or fewer elements. In addition, system 500 may beimplemented using various specific types of hardware components, asappropriate.

B. Screen Projection

Some embodiments may be implemented with a target device that implementsa simple remote viewer and user input handler. Such a configuration doesnot require the target device to provide a browser.

FIG. 6 illustrates a schematic block diagram of a screen projectionsystem 600 used by some embodiments to provide services to a target I/Oelement capable of providing multimedia content. Specifically, thisfigure shows various software components that may be used by the varioussystem elements. Such a software system may be implemented using thehardware system 100 described above. As shown, in addition to (or inplace of) any components described above in reference to system 500,system 600 may include a viewer 610 and system adapter 620 provided by atarget component 120, and a web browser host 630 (with associated API640), third-party 650 and native applications 660, an applicationmanager 670, and local data 680 provided by a host element 140.

In this configuration, the target side may include a command interface530 (e.g., an API) and a simple remote screen viewer 610 that is able todisplay the web applications rendered on the host side. The targetelement 120 may send user inputs to the host 140 and/or may receivescreen bitmaps over the gateway layers 560 of some embodiments. Inaddition, the target 120 may include one or more system adapters 620that may be adapted to retrieve information from an external system(e.g., vehicle information such as location, speed, vehicle status,etc.) and provide the retrieved information to the API 640 (which may beexecuted using a web browser host 630) such that the data may beaccessed by various third party web applications 650 and/or nativeapplications 660.

On the host 140 side, the application manager 670 and the local data 680may be used to manage the third party applications 650 and/or nativeapplications 660. Such applications may be rendered off-screen by a webbrowser running on the host in an appropriate resolution for the target.Any inputs from the target 120 may be received by the gateway 560. Thegateway may also send the rendered screens to the target.

The gateway 560 (or “mobile server”) may be a host component that“wraps” the native web view component to render its content off-screenand send it to the command interface 530 using an efficient remotescreen projection protocol. The command interface may also process theinput from the target and pass the input to the web view. The mobileserver 560 may generate an off-screen canvas that corresponds to thescreen resolution of the target.

One of ordinary skill in the art will recognize that system 600 may beimplemented in various appropriate ways without departing from thespirit of the invention. For instance, different embodiments may includedifferent specific components. As another example, some embodiments mayinclude more or fewer elements. In addition, system 600 may beimplemented using various specific types of hardware components, asappropriate.

C. Vector Screen Projection

Similar to the screen projection configuration described above, someembodiments may be implemented with a target device that is able toreceive, instead of bitmaps of rendered screens, vector informationassociated with, for instance, scaled and calculated HTML pages.

FIG. 7 illustrates a schematic block diagram of a vector screenprojection system 700 used by some embodiments to provide services to atarget I/O element capable of providing multimedia content.Specifically, this figure shows various software components that may beused by the various system elements. Such a software system may beimplemented using the hardware system 100 described above. As shown, inaddition to (or in place of) any components described above in referenceto systems 500 and 600, system 700 may include a specialized viewer 710,a resource cache 720, and a mobile vector service 730.

In this configuration, the host 140 includes the browser 630 and API640, but instead of rendering all the way to a bitmap, the host 140 mayexecute, for instance, JavaScript and transform the vector graphics.Such vector data may be captured by the mobile vector service 730 which,in turn, sends the data to the target 120 using the gateway 560 andcommand interface 530. Similarly, native applications 660 may berendered by the host 140 and individual graphical user interfaceelements may be captured by the mobile vector service 730 and sent tothe target 120.

The target side may include a special viewer 710, which displays thevector graphics received by the host browser 630. The viewer 710 maystore common resources such as graphics, bitmaps, fonts, etc. in theresource cache 720. In this way, less data may be sent over the linkbetween the host 140 and the target 120 and the screens may be renderedfaster. The resource cache 720 may be a local storage that is able tostore common resources to be reused between screen renderings.

The mobile vector service 730 may be a host component that hosts a webbrowser engine. The mobile vector service may intercept the renderingoperations normally performed by the engine and instead of directingsuch operations to bitmap (or screen), the service may construct avector description of the screen and send the description to the target.Such an approach may use an adapter that is able capture the vector datafrom the browser 630. Any resources, such as images, are sent with theirIDs (and/or other appropriate information) and the target 120 maydetermine whether to request the resources if any are not available inthe resource cache 720.

One of ordinary skill in the art will recognize that system 700 may beimplemented in various appropriate ways without departing from thespirit of the invention. For instance, different embodiments may includedifferent specific components. As another example, some embodiments mayinclude more or fewer elements. In addition, system 700 may beimplemented using various specific types of hardware components, asappropriate.

D. Hardware Encoding and Decoding

Some host devices or systems and/or target devices and/or systems mayinclude various hardware components, such as encoders and/or decoders,that may be utilized by some embodiments to perform efficient multimediaprocessing (e.g., rendering) without burdening a general purposeprocessor (and/or other appropriate component). Hardware encoding and/ordecoding may support, for instance, video, audio, generic datacompression and/or communication command processing.

FIG. 8 illustrates a schematic block diagram of a system 800 of someembodiments that utilizes hardware encoding and/or decoding to provideoptimized services to a target I/O element capable of providingmultimedia content. Specifically, this figure shows various softwarecomponents that may be used by the various system elements. Such asoftware system may be implemented using the hardware system 100described above, including sub-elements described in systems 300-400. Asshown, in addition to (or in place of) any components described above inreference to systems 500, 600, and 700 system 800 may include a hardwaredecoder interface 810 and a hardware encoder interface 820. The approachof system 800 may be used, for instance, in conjunction with (and/or inplace of) the various solutions described above in reference to FIGS.5-7.

The hardware decoder interface 810 may be adapted to interact with, forexample, a hardware decoder such as decoder 420 described above inreference to FIG. 4. The hardware encoder interface 820 may be adaptedto interact with, for example, a hardware encoder such as encoder 320described above in reference to FIG. 3.

In this configuration, various rendering (and/or vector processing orother appropriate operations) may be performed using a specializedhardware encoder via the interface 820. Such an approach may allow ahost device to efficiently process the required operation while leavinga processor associated with the host device free (or more free) toperform other system operations. Likewise, a hardware decoder may beused, via the interface 810, to perform various processing operationsusing dedicated decoder hardware such that any processors associatedwith the target element 120 are free (or more free) to perform othersystem operations. In addition, such hardware decoders and/or encodersmay provide enhanced performance (e.g., enhanced speed, powerconsumption, output quality, etc.) compared to the performance achievedwith non-specialized hardware (e.g., processors).

One of ordinary skill in the art will recognize that system 800 may beimplemented in various appropriate ways without departing from thespirit of the invention. For instance, different embodiments may includedifferent specific components. As another example, some embodiments mayinclude more or fewer elements. In addition, system 800 may beimplemented using various specific types of hardware components, asappropriate.

E. Data Flow

FIG. 9 illustrates a data flow diagram of a system 900 including variousdata elements and pathways that may be used by some embodiments.Specifically, this figure shows various dependencies and/or referencesamong various data elements. As shown, the system may include a host 910and a target 920. The host 910 may be any appropriate system or device,as described above in reference to, for instance, FIGS. 1 and 3. Thetarget may be any appropriate system or device, as described above inreference to, for instance, FIGS. 1 and 4. As shown, the host 910 mayinclude a web browser host 925, an OS host 930, one or more other hosts935, a mobile server 940, common resources 945, and a gateway 950. Thetarget 920 may include common resources 955, a communication interface960, a client module 965, a player 970, and a player framework 975.

The web browser host 925 may implement browser hosting and screen (i.e.,content) capturing. The web browser host may use an underlying mobileserver 940 package to send content to a target and/or receive userinputs or commands and simulate the received elements in the browser.

The OS host 930 may be a placeholder for an application host that isable to capture an OS screen. If such a component is needed, it may beOS-specific. The OS host is shown here as an indication of theextensibility of the framework provided by some embodiments.

In addition to the OS host, some embodiments may include other hosts935. Such hosts may be native applications or mobile services that sendscreen, audio and other content to the target using the underlyingmobile server 940 package.

The mobile server 940 may be a package that includes the main code forserving remote applications to a target. The mobile server may beindependent of the actual application providing content. There may bedifferent applications hosts such as a web browser host 925 or an OShost 930 (that intercepts the entire OS), or could be any otherapplication that includes a host component of some embodiments. Themobile server 940 may be able to respond to commands from the target,keep track of target sessions, and/or maintains a state associated withthe target. The mobile server may use the underlying gateway package 950to communicate with the target.

The common resources 945 or 955 may include a package that has commonclasses and code among the target, host, and/or other resources. Thecommon resources may include definitions for messages and events.

The gateway 950 may allow communication among various hosts 910 andvarious targets 920. Each target may likewise include a commandinterface 960 that is able to interact with the gateway 950 in order toexchange data, instructions, user inputs, and/or other elements amongthe host(s) and target(s).

The client module 965 may be a package that represents a target of someembodiments. The client module may be responsible for handling eventsfrom the mobile server 940 and/or sending rendering commands to a playerregister 970. The client module 965 may also receive input events fromthe player 970 and construct messages to the mobile server 940. Theclient module 965 may use the underlying interface 960 to communicatewith the host 910 via the gateway 950.

The player 970 may be an implementation of a Player that is based on anappropriate application and/or UI framework 975 (e.g., the “Qt”framework). There could be other players that utilize differentgraphical frameworks, as appropriate. The player 970 may register withthe client module 965 to receive drawing events and send any inputevents to the player 970.

Although the data flow diagram of FIG. 9 is presented with variousspecific details, one of ordinary skill in the art will recognize thatsuch a system may be implemented in various different ways withoutdeparting from the spirit of the invention. For instance, differentembodiments may include different specific data elements. As anotherexample, various data elements may have different specific relationshipsthan shown (e.g., various data elements may depend on various other dataelements, may be “aware” of other data elements, may include referencesto other data elements, etc.).

F. Exemplary Protocol Commands

Some embodiments may include an API that allows communication amongvarious system elements. The API may be accessible to various devices(e.g., devices having a web browser). Communication may be based oncustom defined messages. Such messages may be carried over anappropriate communication link (e.g., Bluetooth, serial line, WiFi, LAN,USB, etc.) among the nodes (e.g., host, target, server, etc.). Messagesmay not be acknowledged in some embodiments. In some embodiments, nodesmay generate and send a new message in response to a received message.

Messages may include a protocol version included in the message. In someembodiments, nodes (e.g., the server) may only accept incoming commandswhich have the expected version. In some embodiments, the message headermay include a message header size in order to allow backward protocolcompatibility. Each message may include a header and message content.The message header may be implemented by some embodiments as describedbelow in Table 1.

TABLE 1 Parameters: Offset Length [Byte] [Bytes] Type Content Details 01 uint8_t 0x57 First identification byte 1 1 uint8_t 0x47 Secondidentification byte 2 2 uint16_t Command ID Message or command ID 4 4uint32_t Payload size Set to 0 if no additional payload 8

The message content may depend on the message. Various messages thenodes use for communication are described below.

A video configuration command is illustrated below in Table 2. Thecommand may be initiated by the target node to notify the host regardingsupported frame decoders and also the desired frame size and/or format.The host may select and initialize the frame encoder based on theformats supported by the target. Then the host may response with anothervideo configuration command in order to notify the target about theselected frame encoding.

TABLE 2 Message ID: 0x20 Parameters: Offset Length [Byte] [Bytes] TypeContent Details 0 4 Uint32_t Source Suggested view width to be usedwidth by the host. The host may ignore this. 4 4 Uint32_t SourceSuggested view height to be used height by the host. The host may ignorethis. 8 4 Uint32_t Client Width of the frame the host shall width sendto the target. The host should not ignore this. 12 4 Uint32_t ClientHeight of the frame the host height shall send to the target. The hostshould not ignore this. 16 4 Uint32_t Frame The target sends a bit maskwith encoding the supported frame formats. The host shall respond with asingle format. The encoding is as follows: 0x01-I420 + LZ4 0x02-H264 20

A fill rectangle command is illustrated below in Table 3. The commandmay be sent by the host and may include a frame encoded with theselected encoder.

TABLE 3 Message ID: 0x01 Parameters: Offset Length [Byte] [Bytes] TypeContent Details 0 4 int32_t Width Encoded frame width 4 4 int32_t HeightEncoded frame height 8 4 int32_t Encoding Frame encoder ID 12 4 int32_tApp ID Current application ID 16 <var> [byte] Frame bits Encoded framedata 16 + <var>

A setup scroll command is illustrated below in Table 4. The command maybe sent from the host when a new page is opened or connection isestablished. The purpose of this command is to notify the targetregarding the web page extent. The target may decide to setup ascrollbar in order to scroll the web page. The host might not be able toscroll the web page and may need assistance from the target. It is safeto assume that if this command is not sent from the host, then the hostis able to handle the page scrolling alone.

TABLE 4 Message ID: 0x40 Parameters: Offset Length [Byte] [Bytes] TypeContent Details 0 4 Uint32_t Document Page width in pixels. If the pagewidth width is larger than view width, the target should add ahorizontal scroll bar. 4 4 Uint32_t Document Page height in pixels. Ifthe page height width is larger than view width, the target should add ahorizontal scroll bar. 8

A scroll update command is illustrated below in Table 5. This commandcan be sent both from the target and the host. When it is sent by thehost then the target should adjust the scrollbar positions. When it issent by the target then the host should scroll the web page to thespecified position.

TABLE 5 Message ID: 0x41 Parameters: Offset Length [Byte] [Bytes] TypeContent Details 0 4 Uint32_t Horizontal Horizontal scroll positionoffset 4 4 Uint32_t Vertical Vertical scroll position offset 8

A wait indicator command is illustrated below in Table 6. This commandmay be sent by the host to notify the target that some heavy processingis taking place or has been completed. In response the target may, forinstance, show or hide a wait indicator.

TABLE 6 Message ID: 0x15 Parameters: Offset Length [Byte] [Bytes] TypeContent Details 0 1 Boolean State True to show wait indicator, false tohide it. 1

A show keyboard command is illustrated below in Table 7. This commandmay be sent by the host to notify the target that a text element hasbeen selected and some text shall be entered. In response the target maydisplay an appropriate keyboard.

TABLE 7 Message ID: 0x13 Parameters: Offset Length [Byte] [Bytes] TypeContent Details 0 2 Uint16_t Keyboard The requested keyboard type: type0-Text 1-Number 2-Phone 3-email 2

A hide keyboard command is illustrated below in Table 8. This commandmay be sent by the host to notify the target that the keyboard is nolonger necessary. In response the target may close the keyboard.

TABLE 8 Message ID: 0x14 Parameters: Offset Length [Byte] [Bytes] TypeContent Details 0 0 — — No content 0

A mouse command is illustrated below in Table 9. This command may besent by the target in order to notify the host regarding a mouse event(or other similar cursor-control input element) on the target side. Thehost may “playback” the event on the loaded web page.

TABLE 9 Message ID: 0x10 Parameters: Offset Length [Byte] [Bytes] TypeContent Details 0 4 int32_t X position Mouse X position. 4 4 int32_t Yposition Mouse Y position. 8 4 int32_t Mouse Mouse action: action1-mouse moved 2-button down 3-button up 5-button double clicked 12 4int32_t Mouse The mouse button that caused button(s) the action, or thepressed buttons in case of mouse move: 0x00-No button 0x01-Left mousebutton 0x02-Right mouse button 0x04-Middle mouse button 16

A keyboard command is illustrated below in Table 10. This command may besent by the target in order to notify the host regarding a keyboardevent on the target side. The host shall playback the event on theloaded web page.

TABLE 10 Message ID: 0x11 Parameters: Offset Length [Byte] [Bytes] TypeContent Details 0 2 Int16_t Key code The virtual key code 2 4 int32_tKeyboard The keyboard action: action 1-key up 2-key down 6

A browser command is illustrated below in Table 11. This command may besent by the target in order to request from the host a specific browseraction (e.g., go home, go back, go forward, go to specific address,etc.).

TABLE 11 Message ID: 0x12 Parameters: Offset Length [Byte] [Bytes] TypeContent Details 0 1 Uint8_t Browser The requested browser action: action0-Go Back 1-Go Home 2-Go Forward 3-Go to address. In this case theaddress should follow. The length of the address is payload size-1 1<var> [byte] Keyboard The URL as a text. Used only if action the actionis “Go to address”. <var> + 1

A set current application command is illustrated below in Table 12. Thiscommand may be sent by the target in order to request switching to aspecific web application.

TABLE 12 Message ID: 0x42 Parameters: Offset [Byte] Length [Bytes] TypeContent Details 0 4 Uint32_t ID_length Length of application ID 4<ID_length> [byte] App ID Application ID 4 + <ID_length> 4 Uint32_tParams_length Length of application parameters 8 + <ID_length><Params_length> [byte] Application parameters 8 + <ID_length> +<Params_length>

A reconnect command is illustrated below in Table 13. This command maybe sent by the host to notify the target that the host will be stoppedtemporarily. The target may, in response, save the host address and tryto reconnect as soon as possible.

TABLE 13 Message ID: 0x30 Parameters: Offset Length [Byte] [Bytes] TypeContent Details 0 0 — — No content 0

A set frames per second command is illustrated below in Table 14. Thiscommand may be used to dynamically adjust the frame rate depending ontarget load. It may be sent by the target when the rate of receivedframes is greater than or less than the rate of frames the target isable to process.

TABLE 14 Message ID: 0x47 Parameters: Offset Length [Byte] [Bytes] TypeContent Details 0 1 Byte Fps Target frames per second 1

A start audio command is illustrated below in Table 15. This command maybe sent by the host. The command may notify the target that audioplayback is about to start. The command may include informationregarding the audio properties (e.g., codec, sample rate, bits persample, etc.). When this command is received by the target, the targetshould, in response, prepare for the audio playback.

TABLE 15 Message ID: 0x43 Parameters: Offset Length [Byte] [Bytes] TypeContent Details 0 4 Uint32_t Audio The audio codec ID Codec 4 4 Uint32_tSample The audio sample rate, Rate for example 22000. 8 4 Uint32_tSample The sample size in bits, 8 or size 16 12 4 Uint32_t Channel Thenumber of audio channels count 16 4 Uint32_t Byte The audio samples byteorder: order 0-Big endian 1-Little endian 20

A stop audio command is illustrated below in Table 16. This command maybe sent by the host to notify the target that audio playback should bestopped.

TABLE 16 Message ID: 0x44 Parameters: Offset Length [Byte] [Bytes] TypeContent Details 0 0 — — No content 0

An audio data command is illustrated below in Table 17. This command maybe sent by the host. The command may include audio data in a previouslydefined format (e.g., as defined by the start audio command).

TABLE 17 Message ID: 0x45 Parameters: Offset Length [Byte] [Bytes] TypeContent Details 0 <var> [byte] Audio data The audio data <var>

A pause audio command is illustrated below in Table 18. This command maysent by the host to the target when the audio playback should be pausedor resumed. The command may be sent by the target to the host when thehost should pause or resume sending audio data. The target may maintainan audio buffer. When the audio buffer becomes full, a “pause” commandmay be sent. When the buffer is almost empty, a “resume” command may besent.

TABLE 18 Message ID: 0x46 Parameters: Offset Length [Byte] [Bytes] TypeContent Details 0 1 Byte Pause Pause flag: 0-resume 1-pause 1

One of ordinary skill in the art will recognize that the messageelements described above may be implemented in different ways in variousdifferent embodiments. For instance, different embodiments may utilizedifferent specific message IDs. As another example, differentembodiments may include parameters of various different lengths thanthose described above. Furthermore, different embodiments may includevarious additional messages than shown.

III. Application Management

Sub-section III.A provides a conceptual description of a system used bysome embodiments to manage supported applications. Sub-section III.Bthen describes conceptual database schema associated with applicationmanagement.

A. System

FIG. 10 illustrates a schematic block diagram of a back-end system 1000used by some embodiments to manage host applications in someembodiments. As shown, the system may include one or more remote servers1005 and management applications 1010. The back-end system 1000 maycommunicate with a host 1015 running on a mobile device (e.g., asmartphone, tablet, etc.). The host 1015 may be similar to, forinstance, the host 910 described above in reference to FIG. 9.

Returning to FIG. 10, The remote server(s) 1005 may provide a useraccess control (UAC) module 1020, a database access element 1025,storages 1030, an application catalog API 1035, an applicationmanagement API 1040, and a client API 1045.

The UAC module 1020 may determine whether a user has permission toobtain or manipulate a given resource. This module is primarily used bythe application management module 1040. For example, if there is arequest to delete an application, the UAC may check whether the actionis permitted based on the provided user details in the request. Thedatabase access interface 1025 may be generic in order to allow use ofdifferent database management systems. Any appropriate relationaldatabase management system RDBMS may be used. The application managementAPI 1040 may provide a REST API for application management. The API mayallow requests for adding deleting and editing applications. The clientAPI 1045 may provide a REST API that allows information needed for theserver applications to be obtained. Such information may be accessedusing a system access key.

The management applications 1010 may provide an application catalogmanager 1050 and an application manager 1055. The application managermay be a web client that allows users to upload, edit or deleteapplications. The application manager may be implemented as a webapplication that shows the status of the application and allowmanagement of an application approval process. The managementapplications 1010 may be executed using a web browser or nativeapplication that is allowed to interact with the back-end system 1000via the remote server 1005.

The host 1015 may provide a host application 1060, a server library 1065linked to a set of native applications 1070, and a set of HTMLapplications 1075.

The primary role of back-end system 1000 is to provide a list ofavailable applications and provide infrastructure to downloadapplication bundles (e.g., full HTML applications including HTML files,JavaScript files, CSS, resources, etc.) to the host application 1060. Inaddition, the system may expose REST APIs for different web clients thatcan be used to manage application and/or application catalogs. Systemusers may have different permissions over the various APIs.

A given “managed” application (i.e., any application that is managed viathe back-end system 1000 and is able to run on the host 1015) may havemany different forms. The applications could include different types.For example, a given application could be a hosted HTML application onone platform and a native application on another platform. Theapplication could have different styles depending on the connectedtarget system type, selected preferences, and/or other appropriatefactors. The application could have different modes of operationdepending on the connected target system and/or I/O element includedtherein. For example, a high-end vehicle may allow more functions to beenabled compared to a lower-end model. Similarly, there could bedifferent application versions depending on the host platform. Eachindividual incarnation of an application may be defined via anapplication descriptor. There could be many different applicationdescriptors for the same application. Depending on the connected device,the remote server 1005 may send the application descriptor that is themost relevant.

Some embodiments may be able to work with different applications on thehost side. Some may be hosted by the remote server 1005, others may behosted by third party sites or be native apps on the host. Regardless ofthe type, there may be a descriptor for each application stored in theapplication catalog, which may be passed to the host application 1060 onthe host 1015. Supported application types may include, for example,hosted HTML applications, local HTML applications (or applicationbundles), and native applications.

A hosted HTML application may be an HTML application that is hosted onthe remote server 1005 or one or more third party web servers. Theapplication may be provided as an URL to the host application 1060 whichrenders it in the host browser. The application may be partially orfully cached on the host 1015 using a standard HTTP caching mechanism.

A local HTML application may be an HTML application that has all HTML,JavaScript, CSS, and resource files in an application bundle. The bundlemay be downloaded from the remote server 1005 by the host application1060 and may be stored on the host 1015. The application may be loadedin a mobile browser. The application may have a version and the hostapplication 1060 may download a new application bundle if an updatedbundle is available.

Native host applications (e.g., smartphone applications) may be linkedwith the special server library 1065. These applications may beregistered at the remote server 1005 using application identifiers. Thisapproach allows users to select from a main home screen (rendered by thehost application 1060) to switch to native applications using platformspecific methods. Such platform specific methods may include, forexample, using application protocol URLs.

Each application may have many different styles depending on the targetsystem type, selected user preferences, and/or other appropriatefactors. Typically, a style may require “skin” changes without majorfunctionality changes, which may be controlled via an application modeparameter. The styles may be changed dynamically and could be preloaded(pre-cached) on the host. Styles may be primarily CSS and resourcechanges. Examples of styles are “day/night” style, different personalstyles or different vehicle model styles.

The applications could have different functionality based on theselected mode (e.g., style, vehicle type, etc.). The mode may define,for instance, a target type, manufacturer, and/or model that may be usedto enable and/or disable certain functionality based on knowncapabilities of the target. Modes may typically be set upon applicationstartup and not changed dynamically.

A given application may have different implementations depending on thehost platform. For example, an application may be implemented as anative application on one mobile platform and a hosted HTML applicationfor other platforms. This may be reflected via an application platformparameter included in the application descriptor.

Some applications, such as hosted HTML applications, may be generic andrequire some small adaptations to support operation within the system ofsome embodiments. This is achieved via one or more adapter modules (e.g.JavaScript code), which may be injected by the host application into theweb view when an application is loaded. Each adapter module may beassociated with a given application descriptor. In some embodiments, anadapter may be associated with each platform. Alternatively, an adaptermay detect the platform via an agent string and execute the appropriatecode.

In order to support fast application loading and off-line access, someembodiments support local HTML applications. Such applications may beHTML applications, script-based applications, etc. All the applicationfiles (e.g., HTML, JS, CSS, resources, etc.) may be combined into acompressed file and stored on the remote server 1005. Each applicationbundle may have an associated version, which allows the host applicationto detect whether the local bundle needs updating.

One of ordinary skill in the art will recognize that system 1000 may beimplemented in various different ways in different embodiments. Forinstance, some embodiments may include additional modules or fewermodules. As another example, various modules may be combined to form asingle module and/or various single modules may each be divided into aset of sub-modules.

B. Database Schema

The main concept in this database schema is a managed application. Suchan application represented via an application table illustrated below inTable 19. This table describes a managed application. Such anapplication may have a name, description, category, owner, and/or otherelements.

TABLE 19 Field Type Description AppID INT (Primary Key) A uniqueidentifier for the application Name String Name of the application. Thismay be an internal name that uniquely identifies the application. Forexample “XYZ Internet Radio” or “ABC Search App”. The application nameas it will be seen by an end user is defined in the deploymentapplication table. Description String A description of the applicationto help application deployment managers to understand what theapplication does and what the main features are. This information is notshown to the end user and is used primarily for application deploymentmanagers to decide whether to include a given application in adeployment or not. Category String Defines one or more categories inwhich the application belongs. This may be used for easier searching andclassification of applications. IsHomeApp Boolean Flag that defineswhether the application is a home application or not. In a givendeployment there may be only one home application which allows launchingthe other applications. OwnerUserID INT ID of the user who created theapplication. This is a foreign key to a user table.

A given application could have different “incarnations” for differentplatforms, target system models or styles, etc. In some cases theapplication could be an HTML5 application, in others the application maybe native. These variations are described via the application descriptortable illustrated below in Table 20. This table describes applicationdescriptors associated with the application. An application descriptormay refer to a specific application bundle, URL or AppID.

TABLE 20 Field Type Description AppDescriptorID INT (Primary Key) Uniqueidentifier for the application AppID INT Foreign key to the application.DisplayName String Name of the app descriptor. This is used to help theapplication managers select the correct descriptor. The name should beinformative and include the technology and platform. For example “XYZInternet Radio HTML5 Universal” or “ABC Search App Platform DEF Native”.BundleResourceID INT (Optional) Foreign key to the resource descriptortable identifying an HTML application bundle. This may be used with alocal HTML application for the other types of application this field maybe NULL. AdapterMatchRule String (Optional) Rule that identifies how toapply a web application adapter to the current application.AdapterResourceID INT (Optional) Foreign key to the resource descriptoridentifying the application adapter to be applied to the application.The value may be NULL when there is no adapter. URL String URL to ahosted HTML application or application identifier for a nativeapplication. A separate field may be used for native applications.LastModified DateTime Defines when the application was lastupdated/modified. Version String The application version. AppStyleString Defines the styles that the given application supports. These maybe suggestive strings to be used by the application deployment managerto select the correct app descriptor and style. AppMode String Definesthe modes that the given application supports. These are suggestivestrings to be used by the application deployment manager to select thecorrect app descriptor and mode. AppPlatform String Defines the platformthat the given app descriptor supports.

The application may be deployed differently to the various clientsystems. A given deployment may be defined via the deployment descriptortable illustrated below in Table 21. This table may include variousdeployment descriptors. A deployment descriptor may define a specificdeployment configuration that has an associated key. When a hostapplication of some embodiments is executed (e.g., using a smartphone),the key may be passed to the remote server to obtain a list of availableapplications. The back-end system may use the key to find the deploymentdescriptor and then retrieve the catalog of associated deploymentapplications.

TABLE 21 Field Type Description DeploymentDescriptorID INT (Primary Key)Unique identifier for the descriptor. Name String Name of the deploymentconfiguration. This is a name given by the application manager toidentify and distinguish better between the various deploymentconfigurations. Description String Describes the deploymentconfiguration for easier management. WeblinkKey String Alphanumeric keyassociated with the deployment configuration. OwnerUserID INT ID of theuser who created and owns the deployment configuration. This ID maypoint to the user table.

The deployment application table may define the association between aspecific application descriptor and a given deployment. This tableassociates application descriptors with a particular deploymentdescriptor. The table defines, for a given deployment configuration,mode, style and platform, the application that should be shown in thehome screen, using a specific icon and name. The deployments applicationtable is illustrated below in Table 22.

TABLE 22 Field Type Description DeploymentAppID INT (Primary key) Uniqueidentifier for the item. DeploymentDescriptorID INT Foreign key todeployment descriptor. Defines the associated deployment configuration.AppDescriptorID INT Foreign key to application descriptor. Defines thespecific application to be used for the deployment. DeploymentNameString Name of the application as it will be displayed to the user. Mayinclude support for multiples languages. DeploymentIndex INT Order inwhich the application will appear in the home screen. IconResourceID INTForeign key to resource descriptor. Points to the image of theapplication icon to be displayed in the home screen. DeploymentStyleIDINT Foreign key to deployment style. Defines which style the applicationshould use. DeploymentModeID INT Foreign key to deployment mode. Defineswhich mode the application should use. PlatformID INT Foreign key toplatform. Defines which platform the application should use.

Any resources such as images, application bundles, etc. that are storedand managed by the system are represented via the resource descriptortable illustrated below in Table 23. These kinds of resources arereferenced by the other tables by ID. Also, a resource is associatedwith a user who uploaded it and thus may only be available for viewingand editing by authorized users (e.g., users from the same company, asystem administrator, etc.). This table describes a resource such as afile that is managed by the back-end system. The resource could be anapplication bundle, an adapter, an application icon, etc. Each resourcemay be associated with the user who uploaded it.

TABLE 23 Field Type Description ResourceID INT (Primary key) Uniqueidentifier for the resource. UserID INT (Foreign key) The user whocreated and/or uploaded the resource. Points to the user table. AppIDINT Identifies the application with which this resource is associatedwith. ResourceType String The type of resource, e.g. app bundle, appicon, etc. URL String URL of the resource if to be accessed fromoutside. FileName String Name and path of the file as stored on theserver. Size INT Size of the file in bytes. LastModified DateTimeDefines when the resource was last updated/modified.

A platform table, illustrated below in Table 24, defines the availableplatforms for which the system is supported.

TABLE 24 Field Type Description PlatformID INT (Primary key) Uniqueidentifier for the item. FullName String Name of the platform, e.g. ABC4.x Type String The type of platform, e.g. “ABC”, “XYZ”, etc. VersionString Version of the platform, e.g. “4.0.x”

A deployment style table, illustrated below in Table 25, defines theavailable application styles for a given deployment configuration.

TABLE 25 Field Type Description DeploymentStyleID INT (Primary key).Unique identifier for the item. DeploymentDescriptorID INT Foreign keyto deployment descriptor. Defines the associated deploymentconfiguration. IsDefault Boolean Defines whether the style is default ornot. There could be only one default style per given deploymentdescriptor. Name String Name of the style.

A deployment mode table, illustrated below in Table 26, defines theavailable application modes for a given deployment configuration.

TABLE 26 Field Type Description DeploymentStyleID INT (Primary key).Unique identifier for the item. DeploymentDescriptorID INT Foreign keyto deployment descriptor. Defines the associated deploymentconfiguration. IsDefault Boolean Defines whether the mode is default ornot. There could be only one default mode per given deploymentdescriptor. Name String Name of the mode.

The back-end system of some embodiments may be able to be used by manydifferent users that have different privileges. Users are managed by theuser table, illustrated below in Table 27, and user group table.Resources, applications and deployments are associated with users. Thisallows the back-end system to manage who has the privilege to see, editand manage applications or deployments. The user table may includeinformation regarding the users. Users may be associated with onecompany (and/or other similar types of entities) and one or more usergroups. Users from the same company with the same privileges may be ableto access resources created by different users from the same company.

TABLE 27 Field Type Description UserID INT (Primary key) Uniqueidentifier for the item. UserKey String Authentication key for the givenuser. UserGroupID INT Foreign key to user group. Identifies the groupthat the user belongs to. May be implemented as an additional table suchthat one user may belong to multiple groups and one group may havemultiple users. CompanyID INT Foreign key to company table. Identifies acompany associated with the user. AdditionalPermissions String Definespermission for the current user in addition to the permissions from theassociated user group(s). Exclusions String Defines access limitationsfor the current user.

The user group table, illustrated below in Table 28, may define a groupof users with common permission privileges.

TABLE 28 Field Type Description GroupID INT (Primary key) Uniqueidentifier for the item. GroupName String Name of the groupGroupPermissions String Defines the permission rules for the group.

The company table, illustrated below in Table 29, may define a company.One user may be associated with only one company. This may allow usersfrom the same company with the same privileges to be able to accessresources created by other users from the same company.

TABLE 29 Field Type Description CompanyID INT (Primary key). Uniqueidentifier for the item. CompanyName String Name of the companyCompanyDescription String Description of the company.

One of ordinary skill in the art will recognize that the database schemadescribed above are for example purposes only and different embodimentsmay be implemented using various different schema. For instance,different embodiments may use different structures than tables. Asanother example, various different sub-elements may be nameddifferently, included in various different tables, be implemented usingdifferent data types, etc.

IV. Methods of Operation

Sub-section IV.A provides a conceptual description of variouscommunication protocols used by some embodiments. Sub-section IV.B thendescribes various encoding methods used to optimize performance in someembodiments. Next, sub-section IV.C describes various adaptiveadjustments performed by some embodiments. Lastly, sub-section IV.Ddescribes using and target system inputs received through the target.The various methods described below may be implemented using variouscombinations of hardware elements such as those described above inSection I, various combinations of software elements such as thosedescribed above in Section II, and/or other appropriate elements (and/orcombinations of elements).

A. Protocols

Various approaches may be used by some embodiments for remote screencapturing, transmitting and playing at a target. Some embodimentsprovide an extensible protocol that allows for efficient exchange of thedata between the host and the target (and/or between the host and aremote server).

Some embodiments may use a “stateless” implementation where the host isnot aware of the state of the target and waits for requests from thetarget to send screen elements. One example of such an implementation isa traditional VNC protocol. The advantage of such an approach is thatthe host may be simpler, consume less memory, and be able to interactmore easily with multiple targets. However, response latency may be anissue as the target must request an update for the host to send data.Furthermore, bandwidth may be consumed as the host “does not know” whatdata is available to the target and thus must send a full image.

Some embodiments may use a “stateful” implementation where the host isaware of the state of the target and thus may send only changes. Such anapproach reduces the bandwidth requirements as only changes are sent andalso reduces the latency as changes are sent immediately after beingdetected at the host. This approach requires the host to keep a copy ofat least the most recent information send to the target.

A stateful implementation may be the preferred communication mode forsome embodiments, because the stateful mode is intended primarily forpeer-to-peer communication between two devices (e.g., a smartphone andin-vehicle HU). However, the command-based protocol of some embodimentsalso allows mixing stateful and stateless implementations.

A “combinational” implementation may keep state(s) for certain targetdevice(s) and operate statelessly with other targets. Such aconfiguration may be useful, for example, when implementing aconfiguration where a smartphone acts as a server for an in-vehiclehead-unit and also for a set of back-seat entertainment displays.

In order to be extensible and efficient, some embodiments use acommand-based communication protocol. Commands may be sent from the hostto the target with updated screen information (e.g., a new screen, newvideo frame, new vector elements, etc.), or commands can be sent fromthe target with user input information (e.g., touchscreen inputs,keyboard or keypad events, etc.), device information (e.g., screenresolution, capabilities information, etc.), or external systeminformation (e.g., GPS, CAN bus information, etc.). The commands mayinclude header information, properties, and payload with the actualdata.

FIG. 11 illustrates a flow chart of a host-side (or “server-side”)process 1100 used by some embodiments to allow interaction among atleast one host and at least one target. Such a process may begin, forinstance, when a host application of some embodiments is launched by ahost device.

Next, process 1100 may establish (at 1105) a communication link with atarget. Such a link may be established in various appropriate ways(e.g., by automatically transferring a set of messages among the hostand target, based on various user preferences, settings, and/orselections, and/or in other appropriate ways). The operations used toestablish the link may depend at least partly on the type of link (e.g.,a wired connection between host and target, a Bluetooth connection,etc.).

The process may then receive (at 1110) player information from thetarget. Such information may include, for example, screen resolution,screen encoding schema, frame rate, capabilities, etc.

Next, the process may determine (at 1115) whether a command has beenreceived from the target. Such a command may include, for instance, atouch screen command, hardware button press, change in frame rate, etc.In some embodiments, the target device may be a passive device (i.e.,the host may simply send data to the target whether or not any responseis received). In such cases, operation 1120 may be omitted (and/or anyinformation received may be ignored).

If the process determines (at 1115) that no command has been received,the process may continue to operation 1130. Alternatively, if theprocess determines (at 1115) that a command has been received, process1100 may then process (at 1120) the received command and update (at1125) the host state or simulate an input event. In some embodiments,operations 1115-1125 may be iteratively performed as long as thecommunication link is maintained by the host and target.

After determining (at 1115) that no command has been received from thetarget, or after updating (at 1125) the host state, the process maydetermine (at 1130) whether the host state has changed. Such a changemay include, for instance, a change to the screen that requires anupdate (on systems where this may be detected), a new audio frame beingavailable to stream, a screen refresh time has elapsed (e.g., when thehost is not able to determine whether the screen has been changed andsends screen refresh information at regular intervals based on a framerate provided by the target), etc.

If the process determines (at 1130) that the host state has not changed,the process may end. Alternatively, if the process determines that thehost state has changed, the process may then determine (at 1135) whethera stateful implementation is being used. Such a determination may bemade in various appropriate ways (e.g., by querying the target todetermine the capabilities of the target, based on an evaluation of thetype of communication link(s) available, based on the type ofinformation to be sent to the target, etc.).

If the process determines (at 1135) that a stateless implementation isto be used, the process may then prepare (at 1140) a data frame with thecurrent information and continue to operation 1155. Such information mayinclude, for instance, image data, vector data, etc.

If the process determines (at 1135) that a stateful implementation isbeing used, the process may then determine (at 1145) the differencesbetween the previous information and the current information (e.g.,information based on updated data received from a remote server), andprepare (at 1150) a data frame based on the differences.

When preparing (at 1140 or at 1150) a data frame, the process may encodea video frame (e.g., using H264 video encoding format), encode an audiostream (e.g., using mp3 format, advanced audio coding (AAC) format,etc.), reduce the screen rate (e.g., using I420 encoding), and/orperform generic compression of data (e.g., ZLIB compression, LZ4compression, etc.). When using some encoding schema, video data mayinclude only the differences from a previous frame, and thus operation1145 may be omitted in some cases.

After preparing (at 1140 or at 1150) a data frame, the process may send(at 1155) a command including the data frame to the target and then end.In some embodiments, operations 1130-1155 may be iteratively performedas long as the communication link is maintained by the host and target.

In some embodiments, operations 1115-1125 may be implemented in parallelwith operations 1130-1155. In such embodiments, the process may monitorthe target to determine whether any command has been received and alsomonitor the host state to determine whether updates are necessary.

One of ordinary skill in the art will recognize that process 1100 isconceptual in nature and may be implemented in various different wayswithout departing from the spirit of the invention. For instance, thevarious operations of process 1100 may be performed in a differentorder. In addition, various other operations may be performed and/orvarious operations may be omitted, as appropriate. The process may beperformed as part of a larger macro process or divided into multiplesub-processes. Such a process may be performed continuously, at regularintervals, based on satisfaction of a set of conditions, and/orotherwise performed as appropriate.

FIG. 12 illustrates a flow chart of a target-side (or “client-side”)process 1200 used by some embodiments to allow interaction among atleast one host and at least one target. Such a process may begin, forinstance, when a browser or player used by some embodiments is launchedby a target device. Next, process 1200 may establish (at 1210) acommunication link with a host. Such a link may be established invarious appropriate ways (e.g., by automatically transferring a set ofmessages among the host and target, based on various user preferences,settings, and/or selections, and/or in other appropriate ways). Theoperations used to establish the link may depend at least partly on thetype of link (e.g., a wired connection between host and target, aBluetooth connection, etc.).

Next, process 1200 may send (at 1220) player information to the host.Such information may be associated with a player used by the targetdevice to provide content to a user. The player information may includevarious attributes associated with the player (e.g., screen resolution,screen encoding schema, frame rate, capabilities, etc.).

Process 1200 may then determine (at 1230) whether a command has beenreceived from the host. Such commands may use various appropriateformats, protocols, interfaces, etc. A command may be received over thecommunication link.

If the process determines (at 1230) that a command has been receivedfrom the host, process 1200 may process (at 1240) the host command invarious appropriate ways. For instance, such processing may includeframe decoding, drawing of vector elements, processing audio frames,etc.

The host command may include a data frame. Such a data frame may beuncompressed (if compressed), decoded using a software or hardwaredecoder (which may be applied to a previous frame in a statefulimplementation), and/or processed in various other appropriate ways. Thehost command may also include information related toapplication-specific commands (e.g., a notification to show a softwarekeyboard when determining that a user has selected an edit field,notification of an app starting and an associated ID that may be used togenerate a list of active applications and allow a user to select amongthem, etc.).

The process may then update (at 1250) the multimedia output (e.g.,video, audio, graphics, etc.) based on the host command. In someembodiments, operations 1230-1250 may be iteratively performed as longas the communication link is maintained by the host and target.

After updating (at 1250) the multimedia output or after determining (at1230) that no command has been received from the host, the process maydetermine (at 1260) whether a user input has been received. A user inputmay be received through a target element such as a touchscreen, hardwarebuttons, voice input, etc. The user input may be received over thecommunication link. Such a user input may include information related toan application or content being provided on the target device. Forinstance, a user may press a “next” key to skip to the next song in aplaylist, a user may provide input to an application (e.g., an internetradio application) being displayed on the target device, etc.

If the process determines (at 1260) that a user input has been received,the process may prepare (at 1270) a host command and send (at 1280) thecommand to the host. In some embodiments, the command may be encoded atthe target (e.g., audio from a microphone that is encoded into AAC orMP3 frames). Such a command may be formatted in various appropriate waysand include various specific data elements. Such formatting may dependon properties of the target or host, user preferences regarding theoperation of the target or host, etc. The command may be sent over thecommunication link.

After sending (at 1280) the command to the host, or after determining(at 1260) that no use input has been received, the process may end. Insome embodiments, operations 1260-1280 may be iteratively performed aslong as the communication link is maintained by the host and target.

In some embodiments, operations 1230-1250 may be implemented in parallelwith operations 1260-1280. In such embodiments, the process may monitorthe host to determine whether any command has been received and alsomonitor the user inputs of the target to determine whether a user inputhas been received.

One of ordinary skill in the art will recognize that process 1200 isconceptual in nature and may be implemented in various different wayswithout departing from the spirit of the invention. For instance, thevarious operations of process 1200 may be performed in a differentorder. In addition, various other operations may be performed and/orvarious operations may be omitted, as appropriate. The process may beperformed as part of a larger macro process or divided into multiplesub-processes. Such a process may be performed continuously, at regularintervals, based on satisfaction of a set of conditions, and/orotherwise performed as appropriate.

In a similar manner to the approach described above in reference toFIGS. 11-12, a host and remote server (and/or third-party server) mayestablish a communication link and/or send and receive data, asappropriate. In such a configuration, the host and remote server(s) mayperform various different functions (e.g., data storage and retrieval,request handling, etc.) either separately or conjunctively by eachsending and/or receiving data and/or instructions from other systemcomponents.

B. Encoding

In order to achieve optimal throughput, some embodiments use variousencoding methods for encoding data sent to a target. Such an approachmay be primarily used for the screen data sent from the host to thetarget, but can also be used for data sent to the host (and/or remote orthird-party server). Because a command-based protocol is used in someembodiments, the encoding methods may be described in a command headerand may be switched dynamically to adapt to various performanceparameters (e.g., the available bandwidth, application load, etc.)and/or other appropriate factors (e.g., user selection, propertiesrelated to the content, etc.).

FIG. 13 illustrates a flow chart of a process 1300 used by someembodiments to encode data to be sent from a host to a target. Such aprocess may begin, for instance, when data becomes available for a hostto send to a target.

Next, process 1300 may determine (at 1305) whether to perform a screenXOR operation. Such a determination may be made in various appropriateways based on various appropriate factors. If the process determines (at1305) to perform a screen XOR, the process may then perform (at 1310) anXOR operation between a current screen and a most recent screen andencode the changed pixels.

Performing an XOR operation requires a stateful communication mode. Thehost may compare each new screen with the most recently sent screen andsend only the changed pixels. The changes may be encoded using, forexample, run length encoding (RLE) where only the changed pixels aresent and the gaps (i.e., pixels that have not changed) are encoded witha simple count value. Such an approach may useful and very efficient forsmall screen changes.

If the process determines (at 1305) that screen XOR is not to beperformed, or after performing (at 1310) the XOR, the process maydetermine (at 1315) whether to perform a down sample operation. Such adetermination may be made in various appropriate ways based on variousappropriate factors. If the process determines (at 1315) to perform adown sample operation, the process may then down sample (at 1320) thecurrent data.

Typically, a balance needs to be struck between loss of quality, speedof conversion, and image size. For example, one down sampling formatthat some embodiments support is “I420”, which uses twelve bits perpixel and stores the pixels in the YUV color space (using an eight bit Yplane followed by eight bit two-by-two subsampled U and V planes).

If the process determines (at 1315) that a down sample is not to beperformed, or after performing (at 1320) the down sample, the processmay determine (at 1325) whether to perform image encoding. Such adetermination may be made in various appropriate ways based on variousappropriate factors. If the process determines (at 1325) to perform animage encoding operation, the process may then encode (at 1330) thecurrent image data using a standard format.

Such encoding may use standard image formats such as JPG or PNG. JPG canbe quite compact, especially with loss of quality. However, thechallenge is to encode the image fast enough. If done using the host CPU(and if the CPU lacks sufficient processing power), such an approach maybe too slow to achieve twenty-five to thirty frames per second. However,if done using specialized video hardware, the performance could be goodenough to achieve such throughput. Some embodiments support frames orimages encoded with PNG, JPG and/or other formats.

If the process determines (at 1325) that image encoding is not to beperformed, or after performing (at 1330) the image encoding, the processmay determine (at 1335) whether to perform video encoding. Such adetermination may be made in various appropriate ways based on variousappropriate factors. If the process determines (at 1335) to perform avideo encoding operation, the process may then encode (at 1340) thecurrent image data using a standard format.

Standard video encoding formats may provide very good compression andmay be streamed and played back directly by the target. One example ofsuch a video format is “H.264” (or “MPEG-4”). Such an approach requiresthe target to have a video decoder. Another format that may be used is“VP8”, which provides a better encoding speed. Similar to imageencoding, video encoding may be optimized using special video hardware.

The image and video encoding techniques may be used to produce compactand standard image or video streams. Encoding them in real-time may bevery taxing on a host device CPI and there might not be enough CPU powerto achieve a high enough frames per second rate. A solution to thisproblem is to use the specialized video encoding hardware that istypically available on modern devices (e.g., smartphones, tablets,etc.). Many such devices may support real-time video recording, whichtypically uses such fast real-time hardware encoding. Some systems allowaccess to the video encoding hardware and ability to control the inputand output (e.g., via the OS), allowing some embodiments to provide avery efficient and compact remote screen projection solution.

If the process determines (at 1335) that video encoding is not to beperformed, or after performing (at 1340) the video encoding, the processmay determine (at 1345) whether to compress the data. Such adetermination may be made in various appropriate ways based on variousappropriate factors. If the process determines (at 1345) to perform acompression operation, the process may then compress (at 1350) thecurrent data.

Some embodiments may apply a generic compression algorithm to the data.One such option is to use the standard “zlib”, which provides relativelygood compression, but the actual compression process could be very slowand CPU intensive. Another approach is to use the faster “LZ4”compression. Some embodiments support both algorithms, and othercompression types may be added and utilized as well.

If the process determines (at 1345) that compression is not to beperformed, or after performing (at 1350) the compression, the processmay send (at 1355) the data to the target and then end.

One of ordinary skill in the art will recognize that process 1300 isconceptual in nature and may be implemented in various different wayswithout departing from the spirit of the invention. For instance, thevarious operations of process 1300 may be performed in a differentorder. In addition, various other operations may be performed and/orvarious operations may be omitted, as appropriate. The process may beperformed as part of a larger macro process or divided into multiplesub-processes. Such a process may be performed continuously, at regularintervals, based on satisfaction of a set of conditions, and/orotherwise performed as appropriate.

C. Adaptive Adjustment

One of the key features of some embodiments is the use of an adaptiveapproach as to what data is used, how to encode it and what imagequality to use. Because a command-based protocol is used, the commandcan change the format, compression, encoding, frame rate, quality and/orother attributes on the fly.

FIG. 14 illustrates a flow chart of a conceptual process 1400 used bysome embodiments to adaptively adjust various attributes to achieveoptimized throughput and quality based on current operating conditions.Such a process may begin, for instance, when data is available to besent from a host to a target in some embodiments. As shown, the processmay determine (at 1410) available bandwidth (e.g., in frames persecond). Such a determination may be made in various appropriate ways(e.g., by examining data previously sent between the host and target, byretrieving system information that specifies maximum bandwidth, etc.).

Next, the process may determine (at 1420) a main application renderingrate. Such a determination may be made in various appropriate ways(e.g., by retrieving performance information associated with theapplication, by retrieving performance data for one or more hardwareelements used to execute the main application, etc.).

The process may then adjust (at 1430) the frame rate based at leastpartly on the determined parameters (e.g., bandwidth, rendering rate,and/or other appropriate performance parameters). In this way, someembodiments may continuously adjust the frame rate (alternatively, imagequality may be adjust such that smaller or larger data packets aregenerated, as appropriate) based on the determined parameters. Inaddition, the target player may determine the frame rate that the playeris able to support and report the rate to the host via a specialcommand. The host may then set the frame rate of the host as the minimumbetween the preferred frame rate of the target and the possiblesupported frame rate on the host. In this way, the host will not beprocessing and sending more frames than the target can handle.

Next, the process may adjust (at 1440) the screen capture rate based atleast partly on the determined parameters. For instance, screencapturing frequency may be reduced if the main application rendering isslowed down. For example, when a web application is being loaded, thesystem of some embodiments may not need to capture screens, but cannotify the target to wait. This allows the CPU on the host to be free toperform actual application processing.

The process may then adjust (at 1450) image quality based at leastpartly on the determined parameters and then end. As mentioned above,the quality of the images being sent may be reduced when the bandwidthis lower. As another example, lower-quality images may be interspersedamong higher-quality images (e.g., when animation or video is generated,the host may send high-quality images once a second and the rest of theimages may be sent with lower quality to reduce the utilized bandwidth).

One of ordinary skill in the art will recognize that process 1400 isconceptual in nature and may be implemented in various different wayswithout departing from the spirit of the invention. For instance, thevarious operations of process 1400 may be performed in a differentorder. In addition, various other operations may be performed and/orvarious operations may be omitted, as appropriate. The process may beperformed as part of a larger macro process or divided into multiplesub-processes. Such a process may be performed continuously, at regularintervals, based on satisfaction of a set of conditions, and/orotherwise performed as appropriate.

D. User and Target System Inputs

In some embodiments, the target may be responsible for capturing userinput events on the client-side (e.g., a vehicle HU serving as a target)and sending data related to the captured events to a remote server (viathe host) to be processed by a currently running application. Due to theextensible nature of some embodiments, various types of existing orsubsequently developed user input hardware elements (and associatedevents) may be included.

A touchscreen is an example of a hardware element that may be used tocapture user input events in some embodiments. Any captured touch events(e.g., single touch events, multi-touch events, etc.) may be translatedto the host. Hardware buttons (or collections of buttons such askeypads, and/or other appropriate elements) may be translated tohardware key events on the application side (at a host, remote server,and/or third-party server, etc.).

Multiple modes of keyboard input may be supported by some embodiments.For instance, a keyboard may be rendered natively on the target side.The keyboard may be shown when the focus is on an input element. Thisallows for a very responsive UI and a minimum amount of data being sentback and forth, as the keyboard is rendered on the target directly.Alternatively, the keyboard may be rendered by the application itself(e.g., via HTML) and be handled in a similar manner to any other UIelement.

Some target systems (e.g., vehicles) may have special input featuressuch as steering wheel buttons, rotary knobs, etc. Such inputs may besupported through a set of extensible user input commands. The inputsmay be captured by the hardware, and translated by a system adapter(e.g., system adapter 620 described above) into commands and sent to thehost (and/or to a remote or third-party server). On the host and/orserver the commands may be interpreted as generic UI commands (e.g.,scroll, back, etc.) or be handled through a system-specific API.

In addition to user inputs, some target systems (e.g., vehicles) haveother input sources (e.g., sensors related to GPS, speed, etc., CAN orother bus information, etc.) that could be utilized by someapplications. The command structure of some embodiments allow for suchdata to be passed among the target, host, and server. Such data may bepassed as broadcast message, using a subscription-notification feature,using request/response messages, and/or other appropriate ways. Suchinput sources may be handled by a system adapter component.

FIG. 15 illustrates a flow chart of a target-side conceptual process1500 used by some embodiments to capture and react to user input events.Such a process may begin, for instance, when an application is launchedby a target in some embodiments. As shown, the process may receive (at1510) user-generated inputs at the target. The process may then send (at1520) the received inputs to the host. Such inputs may be received,formatted, packaged, and/or sent in various appropriate ways dependingat least partly on the configurations of the target element, the hostelement, the communication pathway between the elements, and/or otherappropriate factors.

One of ordinary skill in the art will recognize that process 1500 isconceptual in nature and may be implemented in various different wayswithout departing from the spirit of the invention. For instance, thevarious operations of process 1500 may be performed in a differentorder. In addition, various other operations may be performed and/orvarious operations may be omitted, as appropriate. The process may beperformed as part of a larger macro process or divided into multiplesub-processes. Such a process may be performed continuously, at regularintervals, based on satisfaction of a set of conditions, and/orotherwise performed as appropriate.

FIG. 16 illustrates a flow chart of a host-side conceptual process 1600used by some embodiments to capture and react to user input events. Sucha process may begin, for instance, when an application is launched by atarget in some embodiments. As shown, the process may receive (at 1610)user-generated inputs from a target. Such user inputs may be receivedfrom a target performing a process such as process 1500 described above.

Next, process 1600 may generate (at 1620) a user input event based atleast partly on the received inputs. The user input event may beprovided to the currently running application on the host (e.g., bysimulating an event on the host), if appropriate.

The process may then generate and send (at 1630) a command to a remoteserver and then end. In some embodiments, the received inputs may beprocessed completely by the host and no command may be sent to theremote server (or vice-versa). For example, user input commands may beprocessed only by the host and sent as user input events to thecurrently active managed application.

One of ordinary skill in the art will recognize that process 1600 isconceptual in nature and may be implemented in various different wayswithout departing from the spirit of the invention. For instance, thevarious operations of process 1600 may be performed in a differentorder. In addition, various other operations may be performed and/orvarious operations may be omitted, as appropriate. The process may beperformed as part of a larger macro process or divided into multiplesub-processes. Such a process may be performed continuously, at regularintervals, based on satisfaction of a set of conditions, and/orotherwise performed as appropriate.

In addition to the operations described above in reference to FIGS.15-16, some embodiments may utilize the host, remote server, and/orthird-party server(s) to react to inputs received from the target. Thehost, remote server, and/or third-party server(s) may be able to,separately or conjunctively, process data and/or instructions to performappropriate actions based on received target inputs. For instance, auser may press a “next” button on a target media player which may causea command to be sent to a host connected to the target. The host mayreceive and interpret the command and generate a request and send therequest to a remote or third-party server. The remote server may executesome action(s) based on the received request (e.g., retrieving dataassociated with a next media element in a list) and provide anappropriate response to the host which may, in turn, send updatedcontent to the target based on the received response.

V. Computer System

Many of the processes and modules described above may be implemented assoftware processes that are specified as one or more sets ofinstructions recorded on a non-transitory storage medium. When theseinstructions are executed by one or more computational element(s) (e.g.,microprocessors, microcontrollers, Digital Signal Processors (DSPs),Application-Specific ICs (ASICs), Field Programmable Gate Arrays(FPGAs), etc.) the instructions cause the computational element(s) toperform actions specified in the instructions.

FIG. 17 conceptually illustrates a schematic block diagram of a computersystem 1700 with which some embodiments of the invention may beimplemented. For example, the system described above in reference toFIG. 1 may be at least partially implemented using computer system 1700.As another example, the processes described in reference to FIGS. 11-16may be at least partially implemented using sets of instructions thatare executed using computer system 1700.

Computer system 1700 may be implemented using various appropriatedevices. For instance, the computer system may be implemented using oneor more personal computers (“PC”), servers, mobile devices (e.g., asmartphone), tablet devices, and/or any other appropriate devices. Thevarious devices may work alone (e.g., the computer system may beimplemented as a single PC) or in conjunction (e.g., some components ofthe computer system may be provided by a mobile device while othercomponents are provided by a tablet device).

As shown, computer system 1700 may include at least one communicationbus 1705, one or more processors 1710, a system memory 1715, a read-onlymemory (ROM) 1720, permanent storage devices 1725, input devices 1730,output devices 1735, various other components 1740 (e.g., a graphicsprocessing unit), and one or more network interfaces 1745.

Bus 1705 represents all communication pathways among the elements ofcomputer system 1700. Such pathways may include wired, wireless,optical, and/or other appropriate communication pathways. For example,input devices 1730 and/or output devices 1735 may be coupled to thesystem 1700 using a wireless connection protocol or system.

The processor 1710 may, in order to execute the processes of someembodiments, retrieve instructions to execute and/or data to processfrom components such as system memory 1715, ROM 1720, and permanentstorage device 1725. Such instructions and data may be passed over bus1705.

System memory 1715 may be a volatile read-and-write memory, such as arandom access memory (RAM). The system memory may store some of theinstructions and data that the processor uses at runtime. The sets ofinstructions and/or data used to implement some embodiments may bestored in the system memory 1715, the permanent storage device 1725,and/or the read-only memory 1720. ROM 1720 may store static data andinstructions that may be used by processor 1710 and/or other elements ofthe computer system.

Permanent storage device 1725 may be a read-and-write memory device. Thepermanent storage device may be a non-volatile memory unit that storesinstructions and data even when computer system 1700 is off orunpowered. Computer system 1700 may use a removable storage deviceand/or a remote storage device 1760 as the permanent storage device.

Input devices 1730 may enable a user to communicate information to thecomputer system and/or manipulate various operations of the system. Theinput devices may include keyboards, cursor control devices, audio inputdevices and/or video input devices. Output devices 1735 may includeprinters, displays, and/or audio devices. Some or all of the inputand/or output devices may be wirelessly or optically connected to thecomputer system.

Other components 1740 may perform various other functions. Thesefunctions may include performing specific functions (e.g., graphicsprocessing, sound processing, etc.), providing storage, interfacing withexternal systems or components, etc.

Finally, as shown in FIG. 17, computer system 1700 may be coupled to oneor more networks 1750 through one or more network interfaces 1745. Forexample, computer system 1700 may be coupled to a web server on theInternet such that a web browser executing on computer system 1700 mayinteract with the web server as a user interacts with an interface thatoperates in the web browser. Computer system 1700 may be able to accessone or more remote storages 1760 and one or more external components1765 through the network interface 1745 and network 1750. The networkinterface(s) 1745 may include one or more application programminginterfaces (APIs) that may allow the computer system 1700 to accessremote systems and/or storages and also may allow remote systems and/orstorages to access computer system 1700 (or elements thereof).

As used in this specification and any claims of this application, theterms “computer”, “server”, “processor”, and “memory” all refer toelectronic devices. These terms exclude people or groups of people. Asused in this specification and any claims of this application, the term“non-transitory storage medium” is entirely restricted to tangible,physical objects that store information in a form that is readable byelectronic devices. These terms exclude any wireless or other ephemeralsignals.

It should be recognized by one of ordinary skill in the art that any orall of the components of computer system 1700 may be used in conjunctionwith the invention. Moreover, one of ordinary skill in the art willappreciate that many other system configurations may also be used inconjunction with the invention or components of the invention.

In addition, while the examples shown may illustrate many individualmodules as separate elements, one of ordinary skill in the art wouldrecognize that these modules may be combined into a single functionalblock or element. One of ordinary skill in the art would also recognizethat a single module may be divided into multiple modules.

It should be understood, of course, that the foregoing relates toillustrative details of exemplary embodiments of the invention and thatmodifications may be made without departing from the spirit and scope ofthe invention as defined by the following claims.

I claim:
 1. An automated method that provides multimedia content to atarget canvas, the automated method comprising: establishing acommunication link from a host mobile device to an external systemassociated with the target canvas; rendering, at the host mobile device,multimedia content for playback by the target canvas, wherein themultimedia content is rendered off screen into screen buffers comprisingsets of vector commands to be transferred to and displayed by the targetcanvas based at least partly on a resolution of the target canvas,wherein the rendering comprises drawing individual graphical userinterface elements and content of each screen of the multimedia contentto an off-screen canvas having a same resolution as the resolution ofthe target canvas, wherein the off-screen canvas is converted into thesets of vector commands by capturing vector information associated withscaled and calculated hypertext markup language (HTML) pages andconstructing a vector description at the host mobile device bytransforming the captured vector information based at least partly onthe resolution of the target canvas; encoding the screen buffers intohost command messages, each host command message comprising a particularencoded screen buffer; and sending the host command messages from thehost mobile device to the external system over the communication link.2. The automated method of claim 1 further comprising: rendering themultimedia content for playback by an alternative target canvas of theexternal system; and sending the rendered host commands to thealternative target canvas over the communication link.
 3. The automatedmethod of claim 1, wherein rendering multimedia content for playback bythe target canvas comprises: determining a bandwidth across thecommunication link; determining a main application rendering rate; andencoding the multimedia content based at least partly on at least one ofthe bandwidth and the main application rendering rate.
 4. The automatedmethod of claim 1, wherein the external system is a vehicle display unitor a medical device.
 5. The automated method of claim 1 furthercomprising: receiving a user input action from the external systemacross the communication link; converting the user input action into aclient command; sending the client command across the communicationlink; receiving the client command at the host mobile device; processingthe client command into a user action; providing the user action to anapplication running on the host mobile device; generating an output ofthe application running on the host mobile device; and modifying therendered multimedia content based at least partly on the output of theapplication running on the host mobile device.
 6. A mobile device thatexecutes web-based applications in a web browser associated with anexternal system, the mobile device comprising: a processor for executingsets of instructions and a non-transitory medium that stores the sets ofinstructions, wherein the sets of instruction comprise: establishing acommunication link between the mobile device and the external system;accessing, at the mobile device, a particular web-based application;rendering, off-screen at the mobile device, a user interface associatedwith the particular web-based application, wherein the renderingcomprises generating screen buffers comprising sets of vector commandsusing a web browser running on the mobile device at an appropriateresolution for the external system by capturing vector informationassociated with scaled and calculated hypertext markup language (HTML)pages and constructing a vector description at the host mobile device bytransforming the captured vector information based at least partly on aresolution of a display associated with the external system; encodingthe screen buffers into host command messages; and sending the hostcommand messages to the external system for display.
 7. The mobiledevice of claim 6, wherein the sets of instructions further comprise:receiving a user input action from the external system; converting theuser input action into a client command; sending the client commandacross the communication link; receiving the client command at a hostmobile device; processing the client command into a user action; andproviding the user action to the particular web-based application at thehost mobile device.
 8. The mobile device of claim 6, wherein thecommunication link with the external system comprises a wirelessconnection.
 9. The mobile device of claim 6, wherein the external systemis a vehicle display unit and wherein the mobile device is a smartphone.10. The mobile device of claim 6, wherein the particular web-basedapplication is accessed via at least one cellular communication network.11. A system that provides multimedia content to a user, the systemcomprising: a target comprising a target display that displaysmultimedia content; and a host that generates multimedia content fordisplay by the target by rendering the multimedia content based at leastpartly on a resolution of the target display, wherein: the multimediacontent is rendered, at the host, into a plurality of screen buffers,each screen buffer including a set of vector commands by capturingvector information associated with scaled and calculated hypertextmarkup language (HTML) pages and constructing a vector description atthe host by transforming the captured vector information based at leastpartly on a resolution of the target display, the screen buffers areencoded into host command messages, the host command messages are sentfrom the host to the target over a communication link, and the targetinteracts with the host across the communication link to at leastpartially control the display of multimedia content.
 12. The system ofclaim 11, wherein the host is a mobile device and the target is avehicle display unit.
 13. The system of claim 11 further comprising aremote server that interacts with the host to at least partially controlthe display of multimedia content, wherein the host and the remoteserver communicate across at least one wireless connection.
 14. Thesystem of claim 11, wherein the host and the target communicate acrossat least one wireless connection.
 15. The automated method of claim 1further comprising: receiving, at the external system, the host commandssent from the host mobile device; and processing the rendered contentincluded in the host commands and displaying the rendered content to thetarget canvas.
 16. The mobile device of claim 6, wherein the sets ofinstructions further comprise: establishing a communication link betweenthe mobile device and a remote server; and accessing, at the mobiledevice, the particular web-based application via the remote server. 17.The system of claim 13, wherein the target comprises at least one userinterface element and wherein: user input actions are received via theat least one user interface element, user input actions are convertedinto client commands, client commands are sent across the communicationlink and received at the host, client commands are processed into useractions, and user actions are provided to the particular web-basedapplication at the remote server.
 18. The automated method of claim 1,wherein the sets of vector commands comprise a fill rectangle commandassociated with a full screen bitmap.
 19. The mobile device of claim 6,wherein the sets of vector commands comprise a fill rectangle commandassociated with a full screen bitmap.
 20. The system of claim 11,wherein the set of vector commands comprises a fill rectangle commandassociated with a full screen bitmap.